Skip to content

Add support for downloading ERA5 pressure levels data product#93

Closed
ewquon wants to merge 55 commits intoNumericalEarth:mainfrom
ewquon:eq/era5_pressure_levels
Closed

Add support for downloading ERA5 pressure levels data product#93
ewquon wants to merge 55 commits intoNumericalEarth:mainfrom
ewquon:eq/era5_pressure_levels

Conversation

@ewquon
Copy link
Copy Markdown
Collaborator

@ewquon ewquon commented Mar 3, 2026

To address #88

ewquon added 10 commits March 3, 2026 11:08
…ressureLevels

  - Updated exports to include ERA5HourlyPressureLevels, ERA5MonthlyPressureLevels, ERA5_all_pressure_levels, pressure_field
  - Added using Statistics, Oceananigans.Fields.CenterField/interior, Oceananigans.BoundaryConditions.fill_halo_regions!, native_grid, InverseGravity to imports
  - Extended import block to include is_three_dimensional, reversed_vertical_axis, conversion_units
  - Added ERA5_all_pressure_levels constant (37 standard hPa levels)
  - Added ERA5PressureDataset, ERA5HourlyPressureLevels, ERA5MonthlyPressureLevels with keyword constructors
  - Added ERA5PressureMetadata{D} and ERA5PressureMetadatum type aliases
  - Added Base.size, all_dates, is_three_dimensional, reversed_vertical_axis dispatches
  - Added ERA5PL_dataset_variable_names and ERA5PL_netcdf_variable_names dicts (15 variables each)
  - Added available_variables, dataset_variable_name, netcdf_variable_name, conversion_units dispatches
  - Added retrieve_data(::ERA5PressureMetadatum) — reads 4D NetCDF, reverses vertical axis
  - Added metadata_prefix(::ERA5PressureMetadata) — uses ERA5PL_dataset_variable_names for filename construction
  - Added _std_atm_geopotential_height, _std_atm_z_interfaces, z_interfaces(::ERA5PressureMetadata)
  - Added pressure_field and mean_geopotential_heights
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
Comment thread src/DataWrangling/ERA5/ERA5.jl Outdated
@codecov
Copy link
Copy Markdown

codecov Bot commented Mar 4, 2026

Comment thread ext/NumericalEarthCDSAPIExt.jl Outdated
Comment thread ext/NumericalEarthCDSAPIExt.jl Outdated
Comment thread ext/NumericalEarthCDSAPIExt.jl Outdated
@navidcy navidcy added the data wrangling 🗃️ JRA55, ECCO, ERA5, and friends label Mar 4, 2026
ewquon added 5 commits March 5, 2026 12:18
Combined CDSAPI requests will return a single combined netcdf; set
cleanup=false to keep the "_tmp_multi_<datetime>_*" files
Note: inpainting is now turned OFF for 2D data -- the reanalysis data
should be complete and turning on inpainting would artificially fill in
data that should be masked (e.g., ocean quantities over land)

Tested for 2D (single levels) and 3D (pressure levels)
to disambiguate from ERA5PressureLevels*
ewquon and others added 9 commits March 25, 2026 13:45
including instantaneous, accumulated, and averaged
…to eq/era5_pressure_levels

Adopt the bbox→region refactor (NumericalEarth#142): rename `metadata.bounding_box` to
`metadata.region` across ERA5 + CDSAPI extension code, examples, and tests.
Drop the obsolete `location(::ERA5Metadata)` override in favor of the new
generic `dataset_location` dispatch, and the redundant per-Metadatum
`metadata_filename` override (the generic version returns the stored
filename). Adopt `reversed_latitude_axis(::ERA5Dataset) = true` for the new
column-field path's φ-coord flip; ERA5's `retrieve_data` overrides keep
their manual `reverse(data, dims=2)` since they bypass the generic path.
Restore the CDS API environment-setup docstring on `download_dataset(::ERA5Metadatum)`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Upstream's column-field test used `ERA5Hourly`, which this branch
renamed to `ERA5HourlySingleLevel` (reserving the simpler name for the
abstract dataset family). Update the test accordingly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ewquon and others added 6 commits April 29, 2026 14:51
Per @glwagner's review: a `Field{Nothing, Nothing, Center}` represents
the same per-level pressure values as the previous `CenterField` but
without copying across the full horizontal grid. Use `set!` for the
column assignment, drop the per-k loop, and let the field's eltype
follow the grid (no more hardcoded `Float32`).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two stopgaps to keep CI green when CDSAPI credentials aren't configured:

- Surface-level tests: extend `download_dataset_with_fallback` with a
  fallback-of-the-fallback. If both CDS and the NumericalEarthArtifacts
  download fail (e.g., the artifact key was renamed when ERA5 dataset
  types were renamed), it now returns `false` instead of throwing.
  Callers `|| return` to skip the testset cleanly with an `@info` rather
  than cascading file-not-found errors.

- Pressure-level tests: two subtests (`Download and 3D Field`,
  `Geopotential height conversion`) still call `download_dataset`
  directly without the fallback wrapper, so guard them with a
  credentials check that early-returns with an `@info`. TODO comment
  references the long-term fix (upload pressure-level test files to
  NumericalEarthArtifacts and switch to the fallback wrapper).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Surface-level testsets each downloaded `2m_temperature` then removed it,
forcing the next testset to re-download the same bytes — three CDS
round-trips for one fixture. Keep the pre-clean only in the testset
that's testing the download path itself; let downstream testsets reuse
the file and run cleanup in the last consumer.

Pressure-level "Geopotential height conversion" pre-cleaned the
`geopotential_...nc` that the previous testset's `z_interfaces`
side-effect leaves on disk, then re-downloaded it. Drop the pre-clean
and the redundant explicit `download_dataset(meta_z)` call (Field()
already downloads if needed).

Net: surface round-trips drop 3 → 1, geopotential round-trips drop 2 → 1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous commit (c6653b2) folded the skip behavior directly into
download_dataset_with_fallback by changing its return type from the
result of download_fn() to a Bool. That broke other callers — JRA55
tests do `fts = download_dataset_with_fallback(...) do ... end;
@test isfile(fts.path)`, which silently NPEs when fts becomes Bool.

Restore download_dataset_with_fallback to its original semantics
(returns download_fn() result, throws on double-failure). Add a new
try_download_or_skip wrapper that catches the double-failure case,
emits @info + CI warning, and returns false — used only by the CDS
surface tests via `try_download_or_skip(...) do ... end || return`.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@ewquon
Copy link
Copy Markdown
Collaborator Author

ewquon commented Apr 30, 2026

@simone-silvestri I'd like to update NumericalEarthArtifacts:

  • Rename 2m_temperature_ERA5Hourly_2005-02-16T12_0.0_5.0_40.0_45.0.nc to 2m_temperature_ERA5HourlySingleLevel_2005-02-16T12_0.0_5.0_40.0_45.0.nc
  • Add temperature_ERA5HourlyPressureLevels_2005-02-16T12_0.0_5.0_40.0_45.0.nc
  • Add geopotential_ERA5HourlyPressureLevels_2005-02-16T12_0.0_5.0_40.0_45.0.nc

Let me know how to proceed.

@giordano
Copy link
Copy Markdown
Member

Note: existing artifacts should never be removed or replaced, otherwise existing tests will fail in the future. For example the renaming operation should be a copy.

@ewquon
Copy link
Copy Markdown
Collaborator Author

ewquon commented Apr 30, 2026

CI passes in #197, closing here.

@simone-silvestri
Copy link
Copy Markdown
Member

Sounds good, maybe we can still add the artifacts to NumericalEarthArtifacts for future users that fork the repo (the instructions should be in the README)

@ewquon
Copy link
Copy Markdown
Collaborator Author

ewquon commented Apr 30, 2026

Will do! Didn't know if I had write access or not.

@ewquon
Copy link
Copy Markdown
Collaborator Author

ewquon commented May 1, 2026

Sounds good, maybe we can still add the artifacts to NumericalEarthArtifacts for future users that fork the repo (the instructions should be in the README)

@simone-silvestri Looks like I don't have push permissions

% gh release upload data-v1 2m_temperature_ERA5HourlySingleLevel_2005-02-16T12_0.0_5.0_40.0_45.0.nc --repo NumericalEarth/NumericalEarthArtifacts
HTTP 404: Not Found (https://uploads.github.com/repos/NumericalEarth/NumericalEarthArtifacts/releases/284970955/assets?label=&name=2m_temperature_ERA5HourlySingleLevel_2005-02-16T12_0.0_5.0_40.0_45.0.nc)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

data wrangling 🗃️ JRA55, ECCO, ERA5, and friends

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants